feat(realtime): add in-memory predicate indexes - #504

Purpose

Linked issue: #158

Add predicate indexes for realtime append-only and deduplicate reads. Queries can use indexed key and value fields to select candidate offsets before reading and filtering rows.

• Introduce pluggable realtime index interfaces and per-field configuration through realtime.index.<plugin>.columns.
• Provide an append-only hash index supporting equality and IN predicates, with concurrent readers sharing postings without copying index data.
• Restrict index queries to [disk snapshot committed end, pinned memory end) and combine candidates with the pinned deletion vector.
• Preserve candidate bitmaps through batch slicing and projection, skip non-matching batches, and evaluate residual predicates only on selected rows.
• Keep memory indexes independent of persisted file indexes. File indexes are built from actual output rows after deduplication, preserving correct file row positions.
• Release pending sealed segments fully covered by a refreshed committed snapshot to reclaim memory before prepare-commit.

Indexes remain in memory when segment data spills and are retained by pinned read views until released.

Tests

Added or extended coverage for:

• Hash postings, offset bounds, concurrent reads, plugin configuration, and conservative predicate fallback.
• Store reads across segment rotation, spill, projection, bitmap slicing, and pinned DV snapshots.
• Predicate evaluation on full, sparse, empty, and nullable selections.
• Realtime integration scenarios covering index pruning without statistics or residual filtering, file-index row remapping after deduplication, snapshot boundaries, pinned plans across updates/deletes and refresh, and residual predicate composition.

API and Format

• Add public RealtimeIndex, RealtimeIndexReader, and RealtimeIndexFactory interfaces.
• Add a required offset_range to RealtimeQueryContext; callers must supply the bounds associated with the pinned memory view and selected disk snapshot.
• No changes to persisted storage formats or protocols.

Documentation

Introduce the optional configuration:

realtime.index.hash.columns=id,payload

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)